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The  FLOW  Tutor:  Schemas  for  Tutoring 

Donald  R.  Gentner  and  Donald  A.  Norman 
University  of  California,  San  Diego 

Consider  now  a human  tutor  might  instruct  a student  learning 
a programming  language.  The  student  attempts  a problem  while  the 
tutor  watches.  The  student  has  difficulty,  makes  some  errors  and 
corrects  some  errors.  Sometimes  the  tutor  gives  advice,  other 
times  the  tutor  simply  waits  for  the  student  to  correct  the 
errors  without  assistance.  All  through  this,  the  tutor  must 
bring  to  bear  a considerable  amount  of  knowledge.  The  tutor  must 
have  a model  of  both  the  student  and  of  the  topic  matter,  simu- 
lating the  progress  of  the  student  through  the  material  to  be 
acquired.  The  tutor  must  consider  the  developing  conceptual 
structures  of  the  student,  the  student's  progress  on  the  current 
task,  and  by  using  some  appropriate  teaching  strategy,  decide 
when  it  is  best  to  intervene  and  when  it  is  best  to  let  the  stu- 
dent work  out  the  problem  alone,  without  assistance. 


Insert  Figure  1 about  here 

Figure  1 shows  a sequence  of  keypresses  from  a student 
attempting  to  solve  a computer  programming  problem.  This  example 
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Figure  1 


Sequence  of  student  keypresses  on  the  comput- 
er terminal.  The  number  of  seconds  since  the 
start  of  the  session  is  indicated  in  the  left 
column . 
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shows  some  of  the  problems  faced  by  an  automated  tutor.  Based 
on  this  information,  an  automated  tutor  must  decide  if  the  stu- 
dent is  making  good  progress,  or  if  not,  what  advice  to  give. 
Human  tutors  can  deal  with  this  situation.  Obviously,  the  human 
tutor  uses  other  information  to  interpret  the  student's 
keypresses:  information  about  the  programming  language  and  the 
particular  way  it  is  implemented  on  this  computer  system,  infor- 
mation about  the  course  of  instruction  and  the  problem  the  stu- 
dent is  attempting  to  solve,  and  information  about  the  knowledge 
structures  of  the  student.  Tutors  must  also  have  a knowledge  of 
learning  principles  to  infer  just  what  course  of  action  would  be 
most  beneficial  for  the  student:  too  much  advice  can  be  as  harm- 
ful as  not  enough.  Moreover,  the  tutor  must  be  flexible.  The 
tutor  must  have  a plan  of  instruction,  but  the  student  may  not  be 
ready  for  that  plan.  The  tutor  must  be  prepared  to  deviate  from 
the  plan  whenever  the  student  behavior  calls  for  new  tactics. 
The  plan  of  the  tutor  constitutes  a top-down,  conceptually  driven 
guidance  of  the  tutorial  session;  the  behavior  of  the  student 
constitutes  a bottom-up,  data  driven  guidance  of  the  session.  A 
successful  tutor  must  be  guided  from  both  directions. 

Our  goal  is  to  understand  the  basic  cognitive  processes 
involved  in  learning  and  teaching,  and  to  develop  computer-based 
theories  and  models  of  these  processes.  Much  of  the  work  has 
been  concerned  witn  the  learning  of  a simple  computer  language, 
known  as  FLOW  (see  Norman,  Gentner  & Stevens,  1976-t- — JAaonan, 


r 
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1975).  FLOW  is  a simple  computer  language  (no  subroutines,  only 
one  variable).  ^ University  undergraduates  with  no  previous 
knowledge  of  computer  programming  can  usually  master  the  basic 
elements  of  FLOW  in  two-to-ten  hours.  In  the  next  section  of  the 
paper,  we  discuss  our  observations  of  human  tutors.  Then,  we 
describe  the  development  of  a schema-based  automated  FLOW  tutor. 

Tutoring 


Insert  Figure  2 about  here 

Figure  2 shows  the  experimental  arrangement.  A student 
sits  in  an  acoustically  isolated  room  with  an  instruction  booklet 
for  the  FLOW  language.  The  booklet  describes  computer  program- 
ming and  introduces  FLOW  with  a series  of  examples  and  program- 
ming problems.  The  student  can  try  out  the  examples  and  attempt 
to  solve  the  problems  on  the  computer  terminal  which  is  connected 
to  a minicomputer.  In  principle  the  student  could  learn  FLOW 
simply  by  reading  the  instruction  booklet  and  trying  out  pro- 
grams on  the  terminal.  In  practice,  however,  students  usually 
have  considerable  difficulty  and  need  advice  from  a human  tutor. 
In  our  most  common  tutorial  arrangement,  the  human  tutor  sits  in 
an  adjacent,  isolated  room  where  a copy  of  the  student's  termi- 
nal screen  is  displayed  on  a TV  monitor.  Any  advice  to  the  stu- 
dent is  relayed  via  a pair  of  linked  terminals,  as  shown  in  Fig- 


ure 2 . 


Observations  of  Human  Tutors 

In  our  studies  of  tutorial  instruction  we  have  used  a 
variety  of  techniques.  Some  have  been  described  previously  (Nor- 
man, Gentner,  & Stevens,  1976;  Gentner,  Wallen,  & Miller,  1974). 
We  examined  a number  of  different  dimensions  of  tutorial  interac- 
tion and  different  methods  of  getting  at  the  conceptual  struc- 
tures of  the  student: 

Tutor  continually  asking  student  to  think  aloud,  or 
simply  observing,  commenting  only  when  necessary; 

A dual-student  interaction,  with  each  student  helping 
the  other,  or  simply  one  student  at  a time; 

Asking  the  human  tutor  to  simulate  an  automated  tutor, 
using  no  more  information  than  would  be  available  to 
such  a system,  or  letting  the  human  tutor  use  every 
source  of  information  possible; 

Watching  students  who  have  no  tutoring  at  all,  but  are 
learning  FLOW  either  in  pairs  or  alone  (in  either  case, 
with  only  the  instruction  manual  and  an  active  terminal 
to  guide  them) ; 


Asking  tutors  to  have  minimum  interaction  or  maximum 
interact  ion ; 
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Retrospective  tutoring,  in  which  an  actual  session  is 
replayed  over  the  computer  terminal  to  the  tutor  who  is 
asked  to  treat  it  as  if  the  student  were  actually  in 
the  adjacent  room.  The  tutor  thinks  aloud,  stating 
what  he  thinks  the  student  is  doing  and  what  advice  he 
would  give,  when,  and  why. 

The  tutors  were  all  experts  at  FLOW,  and  in  many  cases,  experi- 
enced teachers.  The  tutors  had  before  them  a copy  of  the 
instruction  manual  being  used  by  the  students  and  they  were  fami- 
liar with  its  contents  (in  fact,  most  of  the  tutors  had  been 
involved  in  writing  the  manual). 

Some  Comments  on  Exper imental  Procedure 

When  the  tutor  was  in  a different  room  from  the  student,  all 
advice  to  the  student  appeared  on  a second  terminal  located 
beside  the  student.  We  found  it  important  to  use  two  terminals, 
in  order  to  make  a clear  distinction  between  the  tutorial  aspects 
of  the  session  and  the  workings  of  FLOW.  Earlier,  when  we  had 
attempted  to  use  the  same  terminal  for  both  FLOW  and  tutorial 
messages,  we  found  that  the  messages  either  caused  confusion  or 
were  overlooked:  the  students  could  not  distinguish  the  various 
sources  of  information  appearing  on  one  terminal.  Many  students 
have  strange  and  mystical  ideas  about  how  computers  work,  and  the 
use  of  a single  terminal  for  both  functions  added  to  their  confu- 
sion. A separate  terminal,  used  only  as  a tutor,  came  to  be 


viewed  as  the  teacher,  and  students  were  sometimes  recorded  talk- 
ing to  it  ("well,  come  on,  tell  me  what  to  do").  We  believe  the 
advantages  of  using  a separate  terminal  for  the  tutorial  system 
applies  to  most  topics,  including  topics  not  based  around  pro- 
gramming or  use  of  computers. 

We  tried  various  techniques  to  determine  what  the  student 
was  thinking  at  each  point  in  the  session.  One  method  was  simply 
to  have  the  tutor  in  the  same  room  as  the  student,  continually 
asking  the  student  to  "think  aloud,"  or  "what  are  you  thinking 
now,"  or  "why  did  you  do  that?"  In  these  sessions  the  students 
often  seemed  to  be  under  pressure  and  responded  defensively 
despite  our  attempts  to  be  supportive.  Our  best  technique  was  to 
ask  two  students  to  work  together  and  to  put  the  tutor  in  a 
separate  room.  One  student  sat  at  the  FLOW  terminal  and  did 
whatever  typing  was  required.  The  other  student  read  aloud  from 
the  instruction  manual.  The  procedure  was  quite  effective  in 
eliciting  the  thoughts  of  the  students  in  natural  ways,  for  they 
would  discuss  with  each  other  what  they  thought  each  part  of  the 
manual  meant.  When  the  students  encountered  problems,  they  would 
typically  discuss  their  ideas  about  the  source  of  the  problem, 
and  the  possible  alternative  solutions. 

Although  we  tape  recorded  the  comments  of  the  students,  we 
usually  did  not  let  the  tutor  hear  them.  Later,  we  could  replay 
the  session,  allowing  us  to  evaluate  the  tutor's  hypotheses. 


Often  the  students'  comments  showed  that  the  tutor  had  misjudged 


recommend  tnis  two- 


the  problems  faced  by  the  students.  We 
student  instructional  dialog  technique  as  a way  of  getting  natur- 
alistic protocols  of  student  thoughts.  Of  course,  it  is  awkward 
to  attempt  to  simulate  a two-headed  student,  so  for  some  pur- 
poses, the  technique  cannot  be  used.  (We  have  considered  using  a 
trained  experimenter  to  play  the  role  of  the  second  student,  thus 
eliciting  natural  comments  without  the  pressure  of  a tutor's  pry- 

SO 

ing.  There  are  many  difficulties  far  with  this  scheme,  however, 
and  so  we  have  not  used  it.) 

Six  Pr inciplcs  o f Human  Tutor ing 

Our  observations  of  human  tutors  are  hard  to  quantify,  but 
they  did  provide  us  with  useful  principles  with  which  to  guide 
the  development  of  the  automated  tutor.  Human  tutors  are  guided 
in  a variety  of  ways.  The  following  six  principles  emerged  from 
our  observations. 

1:  Conceptually  driven  guidance . Tutors  have  a plan  of 

instruction  which  sets  the  overall  structure  for  the  session  and 
helps  guide  their  expectations  of  student  performance.  This  is 
top-down  or  conceptually  driven  guidance.  Tutors  normally  differ 
in  how  well  formulated  these  instructional  plans  are,  but  in  our 
experiments  the  instruction  manual  was  provided,  so  this  aspect 
of  tutoring  was  determined  beforehand. 

2:  bata-dr iven  guidance . Tutors  are  responsive  to  student 
behavior.  Thus,  they  can  be  data-driven,  deviating  from  the 
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lesson  plan  whenever  it  seems  sensible  to  do  so.  Data-driven 
tutoring  seemed  always  to  be  triggered  by  errors  (which  includes 
a failure  to  make  any  response)  or  by  unexpected  correct 
responses.  The  response  of  the  tutor  to  a student  error  seems 
best  char acter ized  as  an  attempt  to  fina  some  explanation  for  the 
student's  oehavior,  then  postulating  some  instructional  sequence 
that  will  overcome  the  problem.  Thus,  the  events  witnessed  by 
the  tutor  trigger  a search  for  an  adequate  conceptualization  that 
can  serve  as  an  explanation  for  the  observations.  Once  formed, 
the  conceptualization  then  acts  as  top-down,  conceptual  guidance 
for  the  lesson  plan  until  the  tutor  is  satisfied  the  problem  is 
overcome.  Then  the  original  plan  (from  the  instruction  manual) 
is  resumed. 

1:  Active  discovery . Tutors  seemed  unwilling  to  interfere 
too  often.  Their  method  of  instruction  seemed  to  contain  an 
implicit  assumption  that  it  was  best  for  the  students  to  discover 
the  concepts  through  active  exploration,  which  includes  making 
numerous  mistakes.  Tutors  would  therefore  allow  students  to  make 
errors  or  to  pause  for  rather  long  periods  of  time,  offering  help 
only  if  the  number  of  mistakes  or  length  of  pause  exceeded  some 
threshold  value  of  tolerance.  There  were  exceptions  to  this  pol- 
icy. If  an  important  piece  of  information  seemed  to  be  entirely 
missing  then  it  would  be  offered  directly  (to  minimize  time  and 
frustration).  Similarly,  if  the  problem  seemed  unimportant,  it 
didn't  seem  worth  the  student's  effort  to  let  them  worry  aoout 
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it,  and  the  student  was  simply  told  what  to  do  (for  example,  that 
to  type  a quote  mark,  you  must  type  a "shift-2").  Tutors  also 
worried  if  the  student  seemed  to  have  several  simultaneous 
misunderstandings.  The  ideal  situation  for  discovery  learning 
seems  to  be  when  the  student  has  a single  incorrect  concept  and 
the  resources  to  locate  and  modify  tnat  concept  in  a reasonable 
amount  of  time. 

4_:  Say  too  little,  not  to_o  much . There  seemed  to  be  an 
implicit  feeling  among  several  of  the  tutors  that  it  was  best  to 
err  by  saying  too  little  rather  than  too  much.  This  aspect  of 
tutoring  varied  considerably  among  the  tutors,  however.  This  is 
consistent  with  the  principle  stated  above  --  discovery  learning. 
Thus,  the  information  given  the  student  often  was  in  the  form  of 
guides  or  small  fragments  of  information  that  would  allow  the 
students  to  discover  for  themselves  the  total  story.  Sometimes, 
the  advice  was  to  re-read  a section  of  the  manual.  This  form  of 
advice  giving  can  be  characterized  as  tutoring  by  offering  clues 
rather  than  tutoring  by  giving  answers. 

5^  bait  and  see . Tutors  often  hung  back  in  their  assessment 
of  stuaent  difficulty.  This  was  revealed  most  dramatically  when 
we  asked  tutors  to  watch  the  replay  of  previous  learning  ses- 
sions. We  wanted  to  study  how  tutors  updated  their  model  of  the 
student,  so  we  probed  the  tutors  after  each  student  response  (or 
after  each  11  seconds  of  pause).  Tutors  would  often  refuse  to 


speculate,  claiming  that  they  wanted  to  wait  and  see  what  would 
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hat-j-cn  next.  t\e  have  decided  tnat  this  refusal  to  speculate 
after  each  item  of  student  behavior  is  quite  beneficial,  for  it 
limits  the  '.mount  of  needless  search  and  analysis  the  tutor  needs 
to  do.  Sy  waiting  for  a reasonable  sequence  of  responses,  the 
number  of  possible  interpretations  is  considerably  constrained, 
and  the  task  of  forming  a conceptual  model  of  student  performance 
is  much  simplified.  This  strategy  takes  advantage  of  the  fact 
that  it  is  not  necessary  to  correct  a student  problem  the  instant 
it  is  detected. 

b:  h.  little  irrelevancy  is  ok . Tutors  discovered  that  they 
did  not  always  need  to  be  accurate  in  their  assessment  of  student 
behavior.  Oftentimes  a puzzling  sequence  of  behavior  would  turn 
out  to  be  irrelevant,  for  the  student  would  get  on  the  appropri- 
ate track  without  aid.  The  tutor  could  afford  to  ignore  things. 
Even  if  retrospective  analysis  showed  the  behavior  to  be  a hint 
of  forthcoming  difficulties,  little  was  lost  by  not  noting  it 
immediately.  If  there  was  a real  difficulty,  students  would 
reveal  it  again.  Even  when  tutors  misinterpreted  student 
behavior  and  thereby  offered  irrelevant  advice,  little  harm  was 
done:  the  students  seemed  quite  content  to  ignore  the  advice. 
(Unfortunately,  tney  frequently  seemed  content  to  ignore  relevant 
advice,  too.) 

The  above  comments  on  tutorial  strategies  should  not  be  over 
interpreted.  Our  tutors  aid  not  work  independently  of  one 


another.  They  consisted  of  the  several  of  us  who  worked  on  the 
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FLOW  project.  We  often  discussed  our  strategies.  Sometimes 

i 

several  tutors  would  work  together,  so  the  instruction  was  a 
joint  venture  of  two  or  more  people.  Even  when  a single  tutor 
seemed  to  provide  useful  information,  we  have  no  way  of  knowing 
if  this  was  an  optimal  instructional  sequence.  All  we  can  say  is 
that  we  have  observed  ourselves  tutoring  students,  and  we  have 
made  six  generalizations  from  these  observations.  We  will  use 
these  principles  in  the  design  of  the  automated  tutorial  system. 
But  the  studies  are  of  real  tutors,  not  ideal  tutors. 

The  Automated  FLOW  Tutor 

The  automated  FLOW  tutor  is  designed  to  simulate  a human  tutor. 
From  now  on,  unless  we  explicitly  say  "human  tutor,"  the  word 
"Tutor"  will  refer  to  the  computer  program  designed  to  serve 
this  function.  The  Tutor  receives  a message  from  the  student's 
minicomputer  whenever  the  student  presses  a key  on  the  terminal 
or  pauses  for  more  than  about  eleven  seconds.  The  automated 
Tutor  must  interpret  these  keypresses  and  pauses  in  terms  of  a 
student  progressing  through  the  FLOW  instruction  booklet.  If  the 
Tutor  determines  that  the  student  is  having  difficulty,  the  Tutor 
can  send  advice  which  appears  on  a second  terminal  in  the 
stuoent's  room. 

Ambiguities 


Student  responses  are  often  ambiguous.  This  is  especially 
true  where  the  only  information  available  to  the  tutor  is  the 
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time  sequence  of  keypresses  made  by  tne  student.  This,  of 
course,  is  the  only  information  available  to  the  automated  Tutor. 
Ambiguities  can  take  many  forms.  For  example,  in  isolation,  the 
individual  keypresses  are  often  ambiguous.  In  the  FLOW  computer 
system,  when  the  student  types  an  illegal  character,  it  is 
displayed  briefly  accompanied  oy  an  audible  "beep,"  and  then  it 
is  erased.  Determining  why  that  character  was  typed  can  be  dif- 
ficult. But  even  legal  characters  are  often  ambiguous.  In  FLOW, 
for  example,  a D keypress  may  be  part  of  a Display  Quoted  String 
statement.  In  other  circumstances,  a D keypress  could  also  be 
part  of  a Display  Variable  statement  or  part  of  a character 
string.  Similarly,  a two-minute  pause  might  reflect  the  fact 
that  the  student  was  completely  frustrated  and  needed  immediate 
help.  But  a two-minute  pause  could  also  occur  while  the  student 
was  reading  the  instruction  booklet  and  making  perfectly  normal 
progress.  The  task  for  the  Tutor,  then,  is  to  construct  a broad 
and  detailed  model  of  the  student's  conceptual  understanding  and 
activities  and  use  that  model  to  interpret  the  current  behavior 
of  the  student.  This  interpretation  of  the  student's  behavior 
must  then  be  used  to  continually  update  the  model  of  the  student. 

Conceptually  Gu ided  Processing 

The  normal  method  used  to  disambiguate  student  behavior  is 
conceptually  guided  prediction.  The  automated  Tutor  follows  the 
student's  progress  through  the  instruction  booklet,  allowing  for 
pauses  while  the  student  is  reading.  When  an  exercise  or 
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programming  problem  is  encountered,  the  Tutor  does  tne  exercise 
or  solves  the  problem  and  predicts  that  the  student's  actions 
will  follow  a similar  course. 

The  Tutor's  database  contains  a description  of  the  function 
of  each  problem  presented  to  the  student.  The  Tutor  "solves"  the 
problem  by  expanding  the  function  description  into  simpler  func- 
tions, then  into  FLOW  statements,  and  finally  into  the  individual 
keypresses.  For  example,  suppose  that  a problem  at  some  point 
requires  that  the  computer  display  the  word  "BILLY"  on  the 
screen.  The  Tutor  will  eventually  predict  a Display  Quoted 
String  statement  and  that  the  student  will  press  the  D key.  If 
the  student  actually  presses  the  D key  at  that  point,  that  D will 
be  interpreted  as  part  of  a statement  to  display  "BILLY,"  even 
though  it  is  perfectly  possible  at  this  point  that  the  D is  part 
of  some  other  statement.  After  observing  the  D key,  the  Tutor 
will  then  go  on  to  predict  that  the  student  will  press  a quote 
key,  then  a B key,  and  so  forth.  Only  if  these  later  predictions 
are  not  confirmed,  will  the  Tutor  consider  other  possibilities 
for  the  initial  D key.  Although  it  is  clear  that  conceptually 
guided  prediction  can  lead  the  Tutor  into  severe  trouble  when  the 
predictions  are  wrong,  it  normally  leads  to  an  easy  and  efficient 
interpretation  of  otherwise  ambiguous  information. 
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Data-Dr iven  Processing 

Conceptually  guided  processing  works  well  as  long  as  the 
student's  actions  match  the  predictions  of  the  Tutor.  Unfor- 
tunately, students  often  do  unexpected  things.  Remember,  in  our 
observations  of  human  tutors,  we  found  that  they  were  sometimes 
unable  or  unwilling  to  make  detailed  predictions  of  student 
behavior,  especially  when  there  was  more  than  one  reasonable 
alternative  for  the  student.  To  simulate  this  behavior,  the  FLOW 
Tutor  normally  stops  the  conceptually  guided  expansion  of 
instances  when  an  ambiguity  is  reached,  and  waits  to  see  what  the 
student  actually  does  before  going  further.  If  we  call  the  con- 
ceptual structures  that  the  Tutor  uses  to  understand  the  student 
schemas , then  the  job  of  the  Tutor  is  that  of  creating  appropri- 
ate schemas  to  explain  student  behavior. 

The  automated  FLOW  Tutor  is  intended  to  simulate  an  experi- 
enced human  tutor  who  is  familiar  with  FLOW  and  the  problems  stu- 
dents commonly  have  when  learning  it.  The  Tutor's  starting  data- 
base thus  has  schemas  which  represent  typical  student  errors. 
With  data-driven  processing,  unexpected  student  inputs  may 
incorporate  themselves  into  instances  of  these  error  schemas. 
Wnen  one  of  these  error  schemas  is  fully  satisfied,  the  Tutor  has 
effectively  recognized  a student  error  and  can  act  accordingly. 
The  general  strategy  of  the  Tutor  is  to  let  the  student  recover 
from  errors  on  their  own,  and  only  give  advice  when  the  student 
is  likely  to  become  frustrated  or  confused.  Thus,  when  an  error 


-15- 


schema  discovers  a student  error,  it  normally  predicts  that  the 
student  will  pause,  and  only  if  that  pause  is  actually  observed 
will  it  give  advice  to  the  student.  If  the  student  does  some- 
thing before  the  end  of  the  predicted  pause,  the  error  schema 
will  be  unsatisfied  ano  disappear.  The  latest  student  action 
will  initiate  new  schemas  which  will  take  over  the  processing. 
The  pause  length  predicted  depends  on  the  type  of  error  and  the 
Tutor's  moael  of  the  student.  In  general,  if  the  Tutor  believes 
that  the  error  is  related  to  a concept  which  is  new  to  the  stu- 
dent, advice  will  be  given  after  a relatively  short  pause.  On 
the  other  hand,  if  the  student  has  previously  used  this  concept 
correctly,  the  Tutor  will  allow  a longer  pause  before  giving 
advice . 


Schemas 

The  FLOW  tutorial  system  is  based  upon  a schema  representa- 
tion for  knowledge  of  both  the  declarative  aspects  of  the  data 
base  and  the  procedural  aspects  of  tutoring  strategy.  Schema-  or 
frame-based  systems  for  the  representation  of  information  have 
been  proposed  by  numerous  workers  in  Psychology  and  Artificial 
Intelligence,  but  there  have  been  few  attempts  to  use  them  as  the 
basis  for  working  systems  (for  an  example  of  a related  schema- 
based  system  see  Bobrow,  Kaplan,  Kay,  Norman,  Thompson,  & Wino- 


grad-,  1976)  . 
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Insert  Figure  3 about  here 

The  top  left  portion  of  Figure  3 snows  a fragment  of  a FLOW 
program,  consistin'  of  two  Display  Quo  ted  5tr ing  statements. 
Each  statement  has  statement  number  on  the  left  and  the  charac- 
ter string  to  be  displayed  is  included  between  the  quotes.  The 
student  only  has  to  type  the  underlined  characters;  the  other 
characters  are  automatically  provided  by  the  computer.  When 
these  statements  are  executed,  the  words  BILLY  ana  JEAN  would  be 
displayed  on  the  terminal  screen. 

The  general  schema  for  a Display  Quoted  String  statement  is 
shown  in  the  bottom  left  portion  of  Figure  3.  The  notation  fol- 
lows that  of  the  MEMOD  semantic  network  system  described  in  Nor- 
man, Rumelhart  & the  LNR  Research  Group  (1975).  The  schema  has  3 
name,  on  the  top  line,  followed  by  a series  of  slots,  one  on  each 
line.  The  first  two  slots  are  "argument"  slots,  which  are  usea 
to  distinguish  individual  instances  of  the  schema,  and  are  thus 
EMPTY  in  the  generic  schema.  (Our  schemas  typically  have  one  to 
three  arguments.)  The  "specialist"  slot  gives  the  name  of  a pro- 
cedure, in  this  case  DISPLAYQSER,  associated  with  the  schema. 
The  "pnost"  slot  will  be  discussed  later.  The  last  two  slots  in 
the  schema  give  particular  instances  of  this  schema. 

Two  instances  of  the  DI3PLA Y-QDOTED-STRING  schema, 
corresponding  to  the  two  statements  in  the  upper  left  portion  of 
the  figure,  are  shown  on  the  right  in  Figure  3.  The  first  slot 
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on  these  instances  points  back  to  the  original  schema.  This  is 
followed  by  the  two  argument  slots,  which  are  now  filled  in. 
Next  comes  a "status"  slot,  which  indicates  tnat  this  instance 
has  been  OBSERVED.  Schemas  and  instances  are  composed  of  other 
schemas  and  instances,  and  that  structure  is  reflected  in  the 
final  three  slots.  The  "host"  slot  points  to  a higher  level 
instance  whicn  this  instance  is  part  of.  In  these  cases,  the 
instance  is  part  of  a DISPLAY  instance.  Conversely,  the  "ele- 
ment" slots  point  to  the  lower  level  instances  which  make  up  this 
instance.  Here  we  see  that  each  instance  is  composed  of  an 
instance  of  a D schema  and  an  instance  of  a QUOTED-STRING  schema. 
An  instance  may  have  several  elements,  but  is  restricted  to  a 
single  host. 


Insert  Figure  4 about  here 

Figure  4 shows  the  host  and  element  instances  of  the  second 
DISPLAY-QUOTED-STRING  instance  shown  in  Figure  3.  The  DISPLAY 
instance  describes  the  function  of  the  DISPLAY-QUOTED-STRING 
instance,  namely  to  display  JEAN.  It  in  turn  is  part  of  a more 
complex  DISPLAY'-SEQUENCE  instance. 

Tne  D instance  shown  in  the  figure  has  a .'ingle  argument 
slot,  giving  the  time  when  the  student  pressed  this  key. 
Keypresses  and  TIME  messages  are  the  lowest  itvel  schemas  used  by 
tne  FLOw  Tutor,  and  therefore  the  D instance  does  not  have  any 
elements.  The  QUOTED-STRING  instance  shown  on  the  right  also  has 
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a single  argument  slot,  giving  the  value  of  the  string  inside  the 
quotes.  This  instance  is  further  decomposed  into  elements:  two 
instances  of  QUOTE  and  an  instance  of  CHARACTER-STRING. 

All  the  instances  which  the  FLOW  Tutor  creates  and  works 
with  are  part  of  a multi-level  structure.  The  highest  level 
instances  correspond  to  such  things  as  the  instruction  booklet, 
programming  problems,  and  the  function  of  programs.  The  lowest 
level  instances  correspond  to  the  individual  FLOW  statements  and 
keypresses. 

This  hierarchical  structure  of  instances  plays  a major  role 
in  the  operation  of  the  FLOW  Tutor.  Extension  of  the  hierarchy 
to  higher  and  lower  levels  forms  the  basis  for  predicting  and 
interpreting  student  behavior.  The  hierarchical  structure  gives 
the  Tutor  multiple  descriptions  of  the  same  information  at  dif- 
ferent conceptual  levels.  Student  actions  can  then  be  dealt  with 
at  levels  appropriate  for  both  the  Tutor  and  the  student. 

As  a schema-based  system,  the  FLOW  Tutor  has  an  inherently 
distributed  intelligence.  That  is,  there  is  no  central  process 
which  is  "aware"  of  everything  at  a high  level  and  controls  its 
subprocesses.  Instead,  each  schema  knows  only  about  the  things 
which  immediately  concern  it.  When  a schema  and  its  instances 
become  active  they  try  to  find  their  parts  and  fit  themselves 
into  higher  level  schemas.  The  only  central  coordination  is  fur- 
nished by  an  agenda , a simple  list  of  instances  waiting  to  be 
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active.  Distributed  intelligence  systems  such  as  this  are  based 
on  the  premise  that  a large  number  of  relatively  simple, 
independent  processes,  each  concerned  only  with  itself  and  its 
immediate  environment,  will  have  the  net  effect  of  a powerful 
intel 1 igence . 

How  Schemas  Operate 

When  an  instance  becomes  active  in  the  FLOW  Tutor  system, 
the  specialist  for  its  schema  is  invoked.  The  specialist  can 
then  act  based  on  an  examination  of  the  instance  and  any  relevant 
parts  of  the  Tutor's  model  of  the  world.  There  are  several  types 
of  actions  the  specialist  can  perform.  The  specialist  may  modify 
an  instance,  predict  new  instances  of  schemas,  change  the  Tutor's 


model  of 

the 

world,  look 

for 

input 

from 

the 

student,  put 

instances 

on 

the  agenda,  or 

send  messages  to 

the 

student.  In  a 

typical  case, 

an  instance  on 

the 

agenda 

might 

have 

been  predicted 

by  some  other  instance.  When  the  instance  becomes  active,  its 
specialist  would  check  to  see  if  all  of  its  elements  had  been 
observed.  If  not,  the  specialist  would  predict  the  next  element 
and  put  it  on  the  agenda.  If  all  the  elements  of  the  instance 
had  been  observed,  the  specialist  would  change  the  status  of  the 
instance  to  OBSERVED  and  place  its  host  on  the  agenda.  If  the 
instance  had  no  host,  the  specialist  might  search  for  a host  and 
try  to  incorporate  the  instance  into  a suitable  higher  level 


instance . 


A 
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when  a student  responds  in  a manner  not  predicted  by  the 
conceptually  guided  analysis,  the  Tutor  starts  processing  in  a 
data-driven  manner.  When  this  happens,  the  Tutor  stops  the 
conceptuaily-guided  expansion  of  instances  and  waits  to  see  what 
the  student  does.  New  instances  are  made  up  by  lower  level 
instances  or  keypresses,  which  are  initially  without  a host. 
These  instances  try  to  find  a suitable  host,  or  create  one,  and 
the  resulting  structure  builds  up  until  it  joins  some  of  the 
existing  higher  level  structure.  There  are  two  situations  in 
which  data-driven  processing  is  particularly  interesting:  with 
alternate  correct  solutions,  and  with  student  errors. 

A1 ter nate  correct  solutions.  Although  the  programming  prob- 
lems in  the  FLOW  instruction  booklet  are  rather  elementary,  most 
of  them  have  many  different  acceptable  solutions.  When  the  stu- 
dent enters  a program  different  from  that  predicted  by  the  Tutor, 
data-driven  processing  is  used  to  incorporate  the  keypresses, 
statements,  and  simple  functions  into  a representation  for  the 
overall  function  of  the  program.  Information  in  a schema 
representation  is  enmeshed  in  a hierarchy,  and  two  programs  which 
may  be  completely  different  at  the  keypress  or  statement  level 
can  be  equivalent  when  compared  at  higher  levels  which  represent 
simple  or  complex  functions. 

Two  instances  are  considered  formally  equivalent  if  they  are 


instances  of  the  same  schema  and  have  the  same  argument  values; 
they  may  have  different  elements  and  still  be  equivalent.  Thus 
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two  function  instances  might  be  formally  equivalent  even  though 
they  were  composed  of  completely  different  FLOW  statements.  A 
simple  example  of  this  can  be  seen  by  comparing  the  instances  of 
DISPLAY -QUOTED-STRING  ana  DISPLAY  in  Figures  3 ana  4.  The 
DISPLAY-QUOTED-STRING  instance  is  distinguished  in  part  by  its 
statement  number,  but  the  corresponding  argument  slot  for  the 
DISPLAY  instance  gives  its  functional  sequence  in  the  program. 
Thus  the  DISPLAY-QUOTED-STRING  instance  for  the  statement 
026  DISPLAY  "JEAN" 

would  be  distinguished  from  *DISPLAY-QU0TED-STRING-1 932  because 
of  their  different  statement  numbers,  but  at  the  level  of  a func- 
tional description  the  two  DISPLAY  instances  would  have  the  same 
arguments  (both  are  after  *DISPLAY-1853)  and  would  be  formally 
equivalent. 

Handl ing  a Student  Er  ror 


Insert  Figure  5 about  here 

Figure  5 shows  examples  of  error  schemas  operating  within  an 
actual  student  protocol.  The  student  has  written  a program  and 
is  now  about  to  modify  it  to  eliminate  an  error.  The  protocol 
in  Figure  5 begins  as  the  student  is  finishing  a reading  pause. 
The  student  must  list  the  program  before  it  can  be  modified,  and 
the  Tutor  has  therefore  predicted  that  tne  student  will  press  the 
L key  (for  the  List  command) . Instead  of  pressing  the  L key, 
nowever , the  student  presses  the  RUBOUT  key.  RUBOUT  is  an 


i 


00946  TIME 
00954  RUBOUT 
00965  TIME 
00976  TIME 
00987  TIME 

TUTOR:  TO  MODIFY  YOUR  PROGRAM  YOU  MUST  FIRST 
LIST  YOUR  PROGRAM 
00992  RUBOUT 
00999  R 

01010  TIME 

01011  RUBOUT 
01022  X 
01024  D 
01035  TIME 
01046  TIME 

TUTOR:  TO  LIST  YOUR  PROGRAM  YOU  MUST  PRESS 
THE  L KEY 

01058  L 
01068  TIME 
01079  TIME 
01083  RUBOUT 
01088  RUBOUT 
01094  1 
01096  5 
01102  SPACE 
01113  TIME 
01116  D 
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illegal  key  in  this  context,  and  was  not  predicted  by  the  Tutor. 
The  Tutor  now  uses  data-driven  processing  to  interpret  the  unex- 
pected RUBOUT  key.  An  instance  of  RUBCUT  is  created,  and  it 
tries  to  find  a suitable  host  for  itself.  Each  schema  contains 
information  about  possible  hosts  (in  the  "phost"  slot),  and 
RUBOUT  makes  a breadth-first  search  up  through  the  hierarchy 
looking  for  an  instance  which  has  been  predicted.  RUBOUT  eventu- 
ally finds  that  it  could  be  part  of  a predicted  MODIFY-PROGRAM 
instance,  but  MODIFY-PROGRAM  had  predicted  LIST  as  its  next  ele- 
ment. Therefore  RUBOUT  instantiates  the  INCORRECT-ORDER  schema 
on  the  assumption  that  the  student  was  trying  to  modify  the  pro- 
gram, but  forgot  to  list  it  before  using  RUBOUT  to  change  the 
statement  number.  As  is  typical  of  error  schemas,  the  schema  for 
an  INCORRECT-ORDER  error  includes  an  expected  pause.  In  general, 
the  pause  lengths  predicted  depend  on  the  the  type  of  error  and 
the  student's  knowledge  of  the  the  relevant  concepts,  with 
shorter  pauses  allowed  for  concepts  which  are  newer  to  the  stu- 
dent. In  this  case,  since  the  Tutor's  model  of  the  student  indi- 
cates that  this  student  has  already  used  the  List  command,  a 
moderately  long  pause  of  30  seconds  is  predicted.  When  the 
30-second  pause  is  observed,  the  INCORRECT-ORDER  instance  is 
satisfied,  and  the  Tutor  sends  a message  advising  the  student  to 
list  the  program.  Having  just  told  the  student  to  LIST,  the 
Tutor  now  quite  naturally  expects  the  student  to  press  the  L key 
to  list  the  program.  The  student,  however,  tries  several  other 
keys  (most  of  them  illegal)  and  finally,  after  another,  shorter 
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pause,  the  Tutor  gives  more  explicit  advice  to  the  student. 
Current  Status  of  the  FLOW  Tutor 

The  automated  FLOW  Tutor  system  currently  operates  only  in 
"retrospective  mode."  Protocols  are  recorded  as  students  learn 
FLOW,  with  a human  tutor  in  another  room  simulating  the 
automated  Tutor.  (Figure  5 is  an  example  of  such  a protocol.) 
Selected  portions  of  these  protocols  then  serve  as  input  for  the 
automated  Tutor.  . The  system  does  not  yet  have  a sufficient  data- 
base to  follow  a student  through  the  entire  FLOW  course.  In 
addition,  it  is  too  slow  to  respond  in  real  time.  Thus,  the  sys- 
tem has  not  yet  been  used  with  real  students,  although  this  is 
our  eventual  goal.  To  paraphrase  Bobrow  et  al.  (1977),  the  FLOW 
Tutor  does  realistic  tutoring,  but  it  does  not  yet  do  real  tutor- 
ing . 


Summary 

Our  goal  is  to  develop  a theory  of  learning  and  teaching. 
In  particular,  we  are  interested  in  the  process  of  individualized 
instruction  and  in  the  type  of  interactions  which  take  place 
between  a student  and  a teacher.  In  this  paper  we  describe  a 
preliminary  step  towards  the  development  of  such  a theory.  Here, 
we  use  the  observations  of  numerous  tutorial  sessions  between 
human  tutors  and  students  to  characterize  the  tutorial  process. 
Then,  we  show  how  these  ideas  can  be  combined  with  a schema-based 
representational  system  to  form  an  automated  Tutor. 


Three  sources  of  knowledge  must  be  used  in  succesful  tutor- 
ing. First,  there  must  be  knowledge  about  the  subject  matter. 
Second,  there  must  be  knowledge  of  the  student,  including  some 
estimation  of  the  current  state  of  knowledge  that  the  student  has 
about  the  topic  being  studied.  Third,  there  must  be  knowledge  of 
the  principles  of  learning,  so  that  appropriate  tutorial  inter- 
vention can  take  place. 

A successful  model  of  a tutor  must  therefore  contain  all 
these  types  of  knowledge.  In  addition,  the  process  structure  of 
the  tutorial  system  must  allow  it  to  be  both  conceptually  guided 
and  data-driven,  depending  upon  the  performance  of  the  student. 
This  paper  serves  both  as  an  outline  of  some  of  the  properties  of 
tutorial  interaction  and  also  as  a case  study  of  the  development 
of  a working  automated  tutorial  system. 

The  automated  system  cannot  now  instruct  real  students.  At 
the  moment,  the  knowledge  and  processing  structures  required  for 
succesful  tutorial  instruction  in  the  wide  variety  of  situations 
which  can  occur  are  more  complex  than  can  be  handled.  But  we 
believe  the  deficiencies  are  ones  of  quantity,  and  not  of  basic 
principle.  As  computers  become  cheaper  and  more  powerful,  it 
becomes  feasible  to  think  of  small,  independent,  automated  teach- 
ing systems  which  could  have  a real  understanding  of  their  sub- 
ject matter  and  interact  intelligently  with  students  having  dif- 
ferent backgrounds  and  abilities.  Before  such  systems  can  be 
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built  on  more  that  a limited,  ad-ncc  basis,  we  must  learn  a great 
deal  more  aoout  learning,  teaching,  and  tne  representation  and 
organization  of  knowledge.  The  FLOW  Tutor  project  is  directed 
towards  these  goals. 
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Footnotes 

The  research  was  supported  by  the  Advanced  Research  Projects 
Agency  and  the  Office  of  Naval  Research  of  the  Department  of  De- 
fense and  was  monitored  by  ONR  under  Contract  No. 
NG6G14-76-C-G628 . Mark  Wallen  made  significant  contributions  to 
the  research  --  programming  and  maintaining  the  FLOW  system  and 
associated  tutorial  and  analysis  systems,  and  acting  as  tutor  on 
numerous  occasions. 

1.  FLOW  was  originally  developed  by  Jef  Raskin  of  the  Visu- 
al Arts  Department  of  the  University  of  California,  San  Diego, 
specifically  for  students  with  no  mathematics  or  science  back- 
ground; see  Raskin  (1974). 

2.  The  programs  required  of  the  student  are  all  conceptual- 
ly very  simple  --  the  most  complex  problem  given  to  the  student 
is:  "Write  a program  which  will  display  the  word  'yes'  if  the 
input  text  contains  an  E and  'no'  otherwise."  Thus,  the  usual 
difficulties  of  writing  a program  from  a problem  statement  (au- 
tomatic programming)  do  not  arise. 
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